{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "df33c614-2ecc-47a0-8600-bc891681997f",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Profiling Tool for the RAPIDS Accelerator for Apache Spark\n",
    "\n",
    "To run the profiling tool, enter the log path that represents the location of your Spark GPU event logs. Then, select \"Run all\" to execute the notebook. Once the notebook completes, various output tables will appear below. For more options on running the profiling tool, please refer to the [Profiling Tool User Guide](https://docs.nvidia.com/spark-rapids/user-guide/latest/profiling/quickstart.html#running-the-tool).\n",
    "\n",
    "### Note\n",
    "- Currently, local and S3 event log paths are supported.\n",
    "- Eventlog path must follow the formats `/local/path/to/eventlog` for local logs or `s3://my-bucket/path/to/eventlog` for logs stored in S3.\n",
    "- The specified path can also be a directory. In such cases, the tool will recursively search for event logs within the directory.\n",
    "   - For example: `/path/to/clusterlogs`\n",
    "- To specify multiple event logs, separate the paths with commas.\n",
    "   - For example: `s3://my-bucket/path/to/eventlog1,s3://my-bucket/path/to/eventlog2`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Path to the event log in S3 (or local path)\n",
    "EVENTLOG_PATH = \"s3://my-bucket/path/to/eventlog\"  # or \"/local/path/to/eventlog\"\n",
    "\n",
    "# S3 path with write access where the output will be copied. \n",
    "S3_OUTPUT_PATH = \"s3://my-bucket/path/to/output\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "TOOLS_VER = \"24.08.2\"\n",
    "display(Markdown(f\"**Using Spark RAPIDS Tools Version:** {TOOLS_VER}\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install spark-rapids-user-tools==$TOOLS_VER --user > /dev/null 2>&1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "acf401a3-12d3-4236-a6c5-8fe8990b153a",
     "showTitle": true,
     "title": "Environment Setup"
    },
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "# Update PATH to include local binaries\n",
    "os.environ['PATH'] += os.pathsep + os.path.expanduser(\"~/.local/bin\")\n",
    "\n",
    "OUTPUT_PATH = \"/tmp\"\n",
    "DEST_FOLDER_NAME = \"prof-tool-result\"\n",
    "\n",
    "# Set environment variables\n",
    "os.environ[\"EVENTLOG_PATH\"] = EVENTLOG_PATH \n",
    "os.environ[\"OUTPUT_PATH\"] = OUTPUT_PATH\n",
    "\n",
    "CONSOLE_OUTPUT_PATH = os.path.join(OUTPUT_PATH, 'console_output.log')\n",
    "CONSOLE_ERROR_PATH = os.path.join(OUTPUT_PATH, 'console_error.log')\n",
    "\n",
    "os.environ['CONSOLE_OUTPUT_PATH'] = CONSOLE_OUTPUT_PATH\n",
    "os.environ['CONSOLE_ERROR_PATH'] = CONSOLE_ERROR_PATH\n",
    "\n",
    "print(f'Console output will be stored at {CONSOLE_OUTPUT_PATH} and errors will be stored at {CONSOLE_ERROR_PATH}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-10-24T21:27:00.924906Z",
     "iopub.status.busy": "2024-10-24T21:27:00.924587Z",
     "iopub.status.idle": "2024-10-24T21:27:00.928129Z",
     "shell.execute_reply": "2024-10-24T21:27:00.927454Z",
     "shell.execute_reply.started": "2024-10-24T21:27:00.924879Z"
    }
   },
   "source": [
    "## Run Profiling Tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "693b5ee0-7500-43f3-b3e2-717fd5468aa8",
     "showTitle": true,
     "title": "Run Profiling Tool"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "spark_rapids profiling --platform emr --eventlogs \"$EVENTLOG_PATH\" -o \"$OUTPUT_PATH\" --verbose > \"$CONSOLE_OUTPUT_PATH\" 2> \"$CONSOLE_ERROR_PATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f83af6c8-5a79-4a46-965b-38a4cb621877",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Console Output\n",
    "Console output shows the top candidates and their estimated GPU speedup.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c61527b7-a21a-492c-bab8-77f83dc5cabf",
     "showTitle": true,
     "title": "Show Console Output"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "cat $CONSOLE_OUTPUT_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f3c68b28-fc62-40ae-8528-799f3fc7507e",
     "showTitle": true,
     "title": "Show Logs"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "cat $CONSOLE_ERROR_PATH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "05f96ca1-1b08-494c-a12b-7e6cc3dcc546",
     "showTitle": true,
     "title": "Parse Output"
    },
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import shutil\n",
    "import os\n",
    "\n",
    "\n",
    "def extract_file_info(console_output_path, output_base_path):\n",
    "    try:\n",
    "        with open(console_output_path, 'r') as file:\n",
    "            stdout_text = file.read()\n",
    "\n",
    "        # Extract log file location\n",
    "        location_match = re.search(r\"Location: (.+)\", stdout_text)\n",
    "        if not location_match:\n",
    "            raise ValueError(\n",
    "                \"Log file location not found in the provided text.\")\n",
    "\n",
    "        log_file_location = location_match.group(1)\n",
    "\n",
    "        # Extract profiling output folder\n",
    "        qual_match = re.search(r\"prof_[^/]+(?=\\.log)\", log_file_location)\n",
    "        if not qual_match:\n",
    "            raise ValueError(\n",
    "                \"Output folder not found in the log file location.\")\n",
    "\n",
    "        output_folder_name = qual_match.group(0)\n",
    "        output_folder = os.path.join(output_base_path, output_folder_name)\n",
    "        return output_folder, log_file_location\n",
    "\n",
    "    except Exception as e:\n",
    "        raise RuntimeError(f\"Cannot parse console output. Reason: {e}\")\n",
    "\n",
    "\n",
    "def copy_logs(destination_folder, *log_files):\n",
    "    try:\n",
    "        log_folder = os.path.join(destination_folder, \"logs\")\n",
    "        os.makedirs(log_folder, exist_ok=True)\n",
    "\n",
    "        for log_file in log_files:\n",
    "            if os.path.exists(log_file):\n",
    "                shutil.copy2(log_file, log_folder)\n",
    "            else:\n",
    "                print(f\"Log file not found: {log_file}\")\n",
    "    except Exception as e:\n",
    "        raise RuntimeError(f\"Cannot copy logs to output. Reason: {e}\")\n",
    "\n",
    "\n",
    "try:\n",
    "    output_folder, log_file_location = extract_file_info(\n",
    "        CONSOLE_OUTPUT_PATH, OUTPUT_PATH)\n",
    "    jar_output_folder = os.path.join(output_folder,\n",
    "                                     \"rapids_4_spark_profile\")\n",
    "    print(f\"Output folder detected {output_folder}\")\n",
    "    copy_logs(output_folder, log_file_location, CONSOLE_OUTPUT_PATH,\n",
    "              CONSOLE_ERROR_PATH)\n",
    "    print(f\"Logs successfully copied to {output_folder}\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8c65adcd-a933-482e-a50b-d40fa8f50e16",
     "showTitle": true,
     "title": "Download Output"
    },
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "import os\n",
    "import subprocess\n",
    "from IPython.display import HTML, display\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "def display_error_message(error_message, exception):\n",
    "    error_message_html = f\"\"\"\n",
    "    <div style=\"color: red; margin: 20px;\">\n",
    "        <strong>Error:</strong> {error_message}.\n",
    "        <br/>\n",
    "        <strong>Exception:</strong> {exception}\n",
    "    </div>\n",
    "    \"\"\"\n",
    "    display(HTML(error_message_html))\n",
    "\n",
    "def copy_file_to_s3(local_file: str, bucket: str, destination_folder_name: str):\n",
    "    try:\n",
    "        file_name = os.path.basename(local_file)\n",
    "        s3_path = f\"s3://{bucket}/{destination_folder_name}/{file_name}\"\n",
    "        subprocess.run([\"aws\", \"s3\", \"cp\", local_file, s3_path], check=True, capture_output=True, text=True)\n",
    "        return construct_download_url(file_name, bucket, destination_folder_name)\n",
    "    except subprocess.CalledProcessError as e:\n",
    "        raise Exception(f\"Error copying file to S3: {e.stderr}\") from e\n",
    "\n",
    "def get_default_aws_region():\n",
    "    try:\n",
    "        return subprocess.check_output(\n",
    "            \"aws configure list | grep region | awk '{print $2}'\",\n",
    "            shell=True,\n",
    "            text=True\n",
    "        ).strip()\n",
    "    except subprocess.CalledProcessError:\n",
    "        return \"Error: Unable to retrieve the region.\"\n",
    "\n",
    "def construct_download_url(file_name: str, bucket_name: str, destination_folder_name: str):\n",
    "    region = get_default_aws_region()\n",
    "    return f\"https://{region}.console.aws.amazon.com/s3/object/{bucket_name}?region={region}&prefix={destination_folder_name}/{file_name}\"\n",
    "\n",
    "def create_download_link(source_folder, bucket_name, destination_folder_name):\n",
    "    folder_to_compress = os.path.join(\"/tmp\", os.path.basename(source_folder))\n",
    "    local_zip_file_path = shutil.make_archive(folder_to_compress, 'zip', source_folder)\n",
    "    download_url = copy_file_to_s3(local_zip_file_path, bucket_name, destination_folder_name)\n",
    "\n",
    "    download_button_html = f\"\"\"\n",
    "    <style>\n",
    "        .download-btn {{\n",
    "            display: inline-block;\n",
    "            padding: 10px 20px;\n",
    "            font-size: 16px;\n",
    "            color: white;\n",
    "            background-color: #4CAF50;\n",
    "            text-align: center;\n",
    "            text-decoration: none;\n",
    "            border-radius: 5px;\n",
    "            border: none;\n",
    "            cursor: pointer;\n",
    "            margin: 15px auto;\n",
    "        }}\n",
    "        .download-btn:hover {{\n",
    "            background-color: #45a049;\n",
    "        }}\n",
    "        .button-container {{\n",
    "            display: flex;\n",
    "            justify-content: center;\n",
    "            align-items: center;\n",
    "        }}\n",
    "        .button-container a {{\n",
    "            color: white !important;\n",
    "        }}\n",
    "    </style>\n",
    "\n",
    "    <div style=\"color: #444; font-size: 14px; text-align: center; margin: 10px;\">\n",
    "        Zipped output file created at {download_url}\n",
    "    </div>\n",
    "    <div class='button-container'>\n",
    "        <a href='{download_url}' class='download-btn'>Download Output</a>\n",
    "    </div>\n",
    "    \"\"\"\n",
    "    display(HTML(download_button_html))\n",
    "\n",
    "try:\n",
    "    current_working_directory = os.getcwd()\n",
    "    parsed_s3_output_path = urlparse(S3_OUTPUT_PATH)\n",
    "    bucket_name = parsed_s3_output_path.netloc\n",
    "    destination_path = os.path.join(parsed_s3_output_path.path.strip(\"/\"), DEST_FOLDER_NAME.strip(\"/\"))\n",
    "    create_download_link(output_folder, bucket_name, destination_path)\n",
    "    \n",
    "except Exception as e:\n",
    "    error_msg = f\"Failed to create download link for {output_folder}\"\n",
    "    display_error_message(error_msg, e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "73b5e0b0-3a96-4cc6-8e6c-840e4b0d9d43",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "\n",
    "## Application Status\n",
    "\n",
    "The report show the status of each eventlog file that was provided\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c9ffbfdb-dbb6-4736-b9cb-2ac457cc6714",
     "showTitle": true,
     "title": "profiling_status.csv"
    },
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    status_output = pd.read_csv(\n",
    "        os.path.join(jar_output_folder, \"profiling_status.csv\"))\n",
    "\n",
    "    # Set options to display the full content of the DataFrame\n",
    "    pd.set_option('display.max_rows', None)  # Show all rows\n",
    "    pd.set_option('display.max_columns', None)  # Show all columns\n",
    "    pd.set_option('display.width', None)  # Adjust column width to fit the display\n",
    "    pd.set_option('display.max_colwidth', None)  # Display full content of each column\n",
    "\n",
    "    display(status_output)\n",
    "except Exception as e:\n",
    "    error_msg = \"Unable to show Application Status\"\n",
    "    display_error_message(error_msg, e)        \n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "6756159b-30ca-407a-ab6b-9c29ced01ea6",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## GPU Job Tuning Recommendations\n",
    "This has general suggestions for tuning your applications to run optimally on GPUs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "cdde6177-db5f-434a-995b-776678a64a3a",
     "showTitle": true,
     "title": "application_information.csv"
    },
    "jupyter": {
     "source_hidden": true
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    jar_output_folder = os.path.join(output_folder, \"rapids_4_spark_profile\")\n",
    "    app_df = pd.DataFrame(columns=['appId', 'appName'])\n",
    "\n",
    "    for x in os.scandir(jar_output_folder):\n",
    "        if x.is_dir():\n",
    "            csv_path = os.path.join(x.path, \"application_information.csv\")\n",
    "            if os.path.exists(csv_path):\n",
    "              tmp_df = pd.read_csv(csv_path)\n",
    "              app_df = pd.concat([app_df, tmp_df[['appId', 'appName']]])\n",
    "\n",
    "\n",
    "    app_list = app_df[\"appId\"].tolist()\n",
    "    app_recommendations = pd.DataFrame(columns=['App', 'Recommended Configuration'])\n",
    "\n",
    "    for app in app_list:\n",
    "      app_file = open(os.path.join(jar_output_folder, app, \"profile.log\"))\n",
    "      recommendations_start = 0\n",
    "      recommendations_str = \"\"\n",
    "      for line in app_file:\n",
    "        if recommendations_start == 1:\n",
    "          recommendations_str = recommendations_str + line\n",
    "        if \"### D. Recommended Configuration ###\" in line:\n",
    "          recommendations_start = 1\n",
    "      app_recommendations = pd.concat([app_recommendations, pd.DataFrame({'App': [app], 'Recommended Configuration': [recommendations_str]})], ignore_index=True)\n",
    "      html = app_recommendations.to_html().replace(\"\\\\n\", \"<br>\")\n",
    "      style = \"<style>table td { vertical-align: top !important; text-align: left !important; white-space: pre-wrap; } th { text-align: left !important; }</style>\"\n",
    "      display(HTML(html + style))\n",
    "except Exception as e:\n",
    "    error_msg = \"Unable to show stage output\"\n",
    "    display_error_message(error_msg, e) "
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [
    {
     "elements": [],
     "globalVars": {},
     "guid": "",
     "layoutOption": {
      "grid": true,
      "stack": true
     },
     "nuid": "91c1bfb2-695a-4e5c-8a25-848a433108dc",
     "origId": 2173122769183715,
     "title": "Executive View",
     "version": "DashboardViewV1",
     "width": 1600
    },
    {
     "elements": [],
     "globalVars": {},
     "guid": "",
     "layoutOption": {
      "grid": true,
      "stack": true
     },
     "nuid": "62243296-4562-4f06-90ac-d7a609f19c16",
     "origId": 2173122769183716,
     "title": "App View",
     "version": "DashboardViewV1",
     "width": 1920
    },
    {
     "elements": [],
     "globalVars": {},
     "guid": "",
     "layoutOption": {
      "grid": true,
      "stack": true
     },
     "nuid": "854f9c75-5977-42aa-b3dd-c680b8331f19",
     "origId": 2173122769183722,
     "title": "Untitled",
     "version": "DashboardViewV1",
     "width": 1024
    }
   ],
   "environmentMetadata": null,
   "language": "python",
   "notebookMetadata": {
    "mostRecentlyExecutedCommandWithImplicitDF": {
     "commandId": 2173122769183704,
     "dataframes": [
      "_sqldf"
     ]
    },
    "pythonIndentUnit": 2,
    "widgetLayout": [
     {
      "breakBefore": false,
      "name": "Eventlog Path",
      "width": 778
     },
     {
      "breakBefore": false,
      "name": "Output Path",
      "width": 302
     }
    ]
   },
   "notebookName": "[RAPIDS Accelerator for Apache Spark] Profiling Tool Notebook Template"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
